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(True Positive [TP]) and intrusion attempts (False Positives [FP]) by clicking near a 
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1. Introduction 


The defense of computer networks incorporates network monitoring as a critical component for 
which the U.S. Army Research Laboratory (ARL) has become well-respected as result of its in- 
house computer network defense service provider. This network monitoring places heavy 
demands on the human analyst to identify and analyze threats, especially advanced persistent 
threats. Such threats require the analyst to correlate temporally and physically disparate events 
cognitively. The chaotic nature of network traffic data makes it very difficult to differentiate 
normal from malicious traffic. 

Analysts have a difficult task characterized by the need to integrate technical knowledge with 
contextual knowledge under severe constraints. We plan to turn this cognitive overload into an 
opportunity by enhancing the visual displays used by analysts to create tools that more 
effectively reduce the cognitive load while directly aiding the correlation of data through visual 
organization of the data. Analysts typically work with tabular displays or raw data for conducting 
their tasks. Other types of displays providing more representations that are abstract may provide 
more insight into big data inter-relationships, patterns, and areas of interest. 

The goal of this research is to examine and lay out the underlying science and theory of network- 
based intrusion detection—i.e., to develop a rigorous science of intrusion detection. Current 
techniques being developed in a very ad-hoc fashion have very little relevance to the real world 
or any real expectation that the developed techniques will be successful or useful. This can be 
seen in particular with visualization techniques. Numerous visualization techniques have been 
developed over the past decade, but we have not been able to identify any that have ever been 
successfully deployed. It is actually questionable whether analysts have even seen the majority of 
these techniques. Yet new techniques are consistently being designed and published. A further 
complication is the fact that developed techniques are not being tested with real-world data and 
will likely fail in the real world—i.e., with ARL Computer Network Defense Service Provider 
(CNDSP) data. We do not know why these techniques are failing to be deployed: 

• Are analysts being given the opportunity to employ the techniques? 

• Do the techniques meet analysts’ needs or expectations? 

• Do the techniques scale to the size of ARL data successfully? 

• Are the techniques actually identifying relevant malicious activity? 

• Will the techniques reduce analyst time or increase it? 

In response, we have designed a user-study based on a cyber-security analysis game scenario and 
questionnaires to acquire initial insights from real-world analysts. The study acquires the user’s 
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interpretation of display components, captures their cognitive processes as well as contextual 
knowledge, and quantitatively compares tabular versus graphical displays. An additional aspect 
of the study compares real-world analyst feedback with that of students, since students are the 
primary test subjects for academia developing visual displays for network monitoring. In this 
quantitative study, participants act as analysts and their job is to identify as many of the network 
threats on the simulated network provided. We will observe the participants’ responses to the 
pattern-matching activity created within the game scenario. The design variables will be the 
distinct graphical layouts. The response variables are true-positive and false-positive rates of 
event identification, the time required for event identification, and the qualitative questionnaire. 
Results will help us understand which of the visual layouts is most effective for predicting cyber 
attacks. This will benefit network security analysts who defend the nation’s networks. 


2. Information Visualization 


Automated systems requiring vigilant human insight are one potential solution to combat 
computer security threats. It is recommended that these systems incorporate a human in the 
diagnostic loop since his/her analytic skills far surpass that of computers (2). In general, support 
tools are needed to integrate intricate sense-making capabilities with the ability of these 
automated systems to process vast quantities of data ( 4 ). Information visualization is defined { 28 ) 
as a computer-supported, interactive visual representation of data to amplify cognition. 
Information visualization is one such method that shows great potential for supporting computer 
security work in that it provides the human security analysts with better tools to discover 
patterns, detect anomalies, identify correlations, and communicate findings, all while keeping the 
human in the diagnostic loop. Information visualization can be used for exploration discovery, 
decision-making, and communication of complex ideas, and it helps to deal with processing the 
influx of data. This is an interactive method used to represent abstract data when compared to 
other data graphics. Information visualization tools allow the user to adjust the display in order to 
gain a more meaningful understanding of the data being presented. Mapping the data spatially in 
a meaningful matter is the most important and challenging part to making an effective 
information visualization (4). At the core of information visualization is the goal of amplifying 
cognition, the intellectual processes in which information is obtained, transformed, stored, 
retrieved, and used (d). Robust information visualization tools that implement the importance of 
keeping humans in the loop take advantage of the power of the human perceptual and cognitive 
processes in solving computer security problems. 

2.1 Visual Representation in Table Form 

Analysts are used to, and most times prefer, tabular displays. Tabular displays originate from 
spreadsheet techniques that provide a structured, intuitive, and powerful interface for 
investigating information visualizations of multidimensional datasets (5). Mathematicians and 
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statisticians have long used tables of sine, cosine, and confidence probabilities. Previously, the 
invention of the VisiCalc numerical spreadsheet in 1979 fueled the adoption of usage with 
personal computers (d). Statisticians have examined visualizing higher dimensional point sets by 
a table of projections. For example, one multivariate analysis tool is the scatter matrix, which is a 
table of scatter plots (7). Since the early 1980s, visualization researchers have applied similar 
ideas, but in different ways, to produce a table of views of a single dataset (8, 9). These 
approaches represent a largely static tabular approach to the data, but some interactivity is 
present, such as rotations, translation, and zooming. There are several distortion presentation 
techniques based on a tabular layout (JO) such as Document Lens (JJ), fish-eye views (J2, 13), 
stretching rubber sheets (14). Overall, the advantages for analysts using the tabular layout are 
that it is familiar, flexible, easily configurable, and excellent for interactive comparison tasks (5). 

2.2 Visual Representation in Graphical Form 

We recommend using graphical representations to illustrate network activity and relationships 
among network components. This study is an approach toward providing analysts with enhanced 
visual displays that will ease their difficult task of integrating technical knowledge with 
contextual knowledge under severe constraints. Much research has been done to identify and 
develop external aids that enhance cognitive abilities for end-users. Visualization, itself, has been 
identified as a necessary and effective technology for network security, particularly, with 
intrusion detection systems (IDS) (26). While information visualization remains a novelty for 
some users, who struggle to use the graphics effectively, this study’s suggested graphical 
representations of network data highlight components, patterns, relationships, and features that 
increase the utility of user displays and the likelihood of adoption by industry. 

Various workflow visualization tools are available to help users track their analysis, reuse 
effective workflows, and test hypotheses (1). However, the need still exists for analysts to 
improve communication and performance, explore deeper into certain network attacks, and 
investigate suspicious activities within a network (27). Some past visualization techniques have 
contributed to better visual displays for end-users: 

Flow-Based approaches (HistoryFlow, ThemeRiver, TimeWheel, Wormplots) (28-31) 

* Glyph-Based approaches (32, 33) 

• Circle Segment (34, 35) 

Our approach to addressing analyst’s needs for visualization information differs from previous 
works in two ways: the tools used and the focus of what is being visualized. In our case, we plan 
to turn visual overload into an opportunity by enhancing the visual displays used by analysts into 
a more effective tool. Traditionally, analysts are used to working with tabular displays for 
conducting their tasks. Other types of displays, such as a graphical display, may provide more 
insight into big data inter-relationships, patterns, and finding areas of interest. In response, we 
have designed an experiment using cyber-network data to test the effectiveness for 
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communicating suspicious activity on a computer network through visual displays. In the study, 
participants act as analysts, and their job is to identify as many as possible of the intrusion 
attacks and intrusion attack attempts on the tabular and graphical displays provided. The design 
variables will be several distinct graphical layouts. The response variables are true-positive and 
false-positive rates of event identification, the time required for event identification, and a 
qualitative questionnaire. Results will help us understand which of the visual layouts is most 
effective for predicting cyber attacks. This will benefit network security analysts who defend the 
nation’s networks. 


3. User Studies 


Evaluating scientific visualization techniques is a longstanding challenge (15-17). Similarly, the 
field of information visualization has a strong tradition in pioneering research in evaluation 
techniques (18-20). User studies often rely on timing and accuracy information collected during 
the study, coupled with subjective user surveys given after the experiment is completed. This 
combination of empirical measurement with a subjective questionnaire is designed to assess the 
efficacy of a visualization technique with respect to related methods. However, the analysis of 
user evaluation studies remains difficult. These challenges are often compounded by the limited 
empirical data acquired during the study. Beyond the specific details of the many user study 
experiments, they all share a common goal: to assess the strengths and weaknesses inherent to a 
visualization technique or system. Incorporating as many objective measures as possible into the 
experiment not only provides a more robust analysis, but also mitigates subjectivity often 
introduced by users’ preferences, biases, and retrospection. In this position paper, we review 
traditional evaluation techniques that consist of data gleaned from system logging. We then 
outline evaluation methods using physiological measures for the assessment of scientific 
visualization efficacy. 

Due to the nature of today’s complex scientific data, simply displaying all available information 
does not adequately meet the demands of domain scientists. Determining the best use of 
visualization techniques is one of the goals of scientific visualization evaluations. The types of 
improvements offered by the method being studied dictate evaluation methods. Some evaluations 
are concerned primarily with technological improvements, such as rendering speed or the 
management of large data. User studies have been used to evaluate everything from aircraft 
cockpits (21) and surgical environments (22) to visualization methods (23). Evaluating 
visualization methods that focus on human factors often employ user studies or expert 
evaluations to determine their effects on interpretation and usability. An expert assessment takes 
advantage of knowledgeable users to enable more poignant analysis of use cases, and these 
experts also bring their own preconceptions and preferences that can skew studies. Traditional 
evaluation methods provide mechanisms to gauge aspects of visualizations or environment. 
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Unfortunately, experiments using surveys to measure user experience introduce subjectivity and 
bias from the users. Subjectivity in user responses may be partially mitigated using 
questionnaires developed with the Likert Scale (24). Subjectivity in evaluation may provide 
important insights into how users interact with the systems being studied. However, subjective 
measures do not help answer questions regarding how effective a method is at eliciting insight 
from a dataset. This is a primary purpose of visualization. Our goal and purpose is to use this 
project as an empirical study to examine the cognitive aspects of visual displays, with the goal of 
identifying components and representations that most effectively aid the computer network 
analyst in interpreting the underlying activity in a network sample. Results from the study are 
helpful to understand the potential and limitations of the suggested visual displays attempting to 
aid analysts’ needs to better achieve their tasks. 

3.1 Study Development 

Step 1: Performed literature review on the following topic areas: 

• Existing Visual Representations: 

o What current visual representations exist that could be applied to analysts’ displays? 

• Visualization Tools: 

o What visualization tools are currently being used for analysts’ displays? 
o What is it about the tools that work for analysts and what analysts needs remain unmet 
by these tools? 

o Are there other visualization tools not specific to the network domain that could be of 
use for analysts display visualization needs? 

• Existing User Studies 

o What studies have been done with visualization displays? 
o What studies have tested analysts’ displays? 

• Consider New Methods for Displays 

o What visualizations have been effective on displays used in other domains (medical 
field, biology field, etc.)? 

Step 2: Completed the following Human Eactors Trainings: 

• The Principal Investigators (Pis) had to complete the Collaborative Institutional Training 
Initiative (CITI) at https://www.citiprogram.org and score at least 80% on each exam. 

• The Participants Investigators (Pis) had to also complete the National Institutes of Health 
(NIH) at http://phrp.nihtraining.com/ and pass each exam. 

Step 3: Developed a protocol for the study by the following: 

• The principal investigators held meetings to discuss parameters and theory of the study: 

Research resulted in an experimental design for the study. Generally, the study is broken 
into two parts: a preliminary study that uses graphical methods to present network 
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information to users on a display, and a follow-up study that uses a game scenario to 
present the same displays but allows user interaction with the visual representations in the 
game for data exploration of interesting features. The preliminary study compares graphical 
methods to tabular displays typically used in real network analyst environments. There are 
two phases for the study. The objective of Phase 1 is to evaluate how the users’ abilities to 
detect network “intrusions and possible intrusions” is affected by the three display 
strategies. In the preliminary study in Phase 1, information will be presented in static 
displays, and in the follow-up study in Phase 1, information will be presented by the CyFall 
game. In Phase 2, an emphasis will be placed on understanding analysts’ cognitive 
processes. Phase 2 uses a reasoning support system developed by Penn State University to 
assist the analyst in formulating hypotheses about the state of a network. The steps the 
analysts engage in to formulate and discard hypotheses will be recorded. 

• Collaboration: 

ARL’s Computational and Information Sciences Directorate (CISD) teamed up with the 
Human Research and Engineering Directorate (HRED) to form and conduct the study. We 
invited Morgan State University (MSU) student subjects are participants in the study; 
contracted Stony Brook University to manufacture software for the cyber-network game 
scenario, CyEall; and involved Pennsylvania State University (PSU), who produced the 
trace software to capture analysts’ cognitive processes. 

• Subjects: 

This study compares the results of expert analysts with that of university students since 
university students are the primary test subjects for academia developing visual displays for 
network monitoring. The expert analysts come from ARL’s Sustaining Base Network 
Assurance Branch (SBNAB) team at both the Adelphi Laboratory Center (ALC) and 
Aberdeen Proving ground (APG) locations. The university students come from MSU. 

• Eormulating Questionnaires: 

o Demographic questions were created to establish the census of the subjects 

participating in the study. Questions inform of confidence level with pattern-matching 
activities, prior experience in analysts tasks, and into what populations they fall 
(male/female, age, etc.). 

o Pre-task questions were created to measure the subject’s subjective perception of 
representations. 

o Post-task questions were created to determine performance satisfaction, and to gather 
the overall aptitude of the tools used, visual representations seen, game environment 
response, and special user insights. 

We implemented the questionnaires in a Web-based open source survey application called 
LimeSurvey (36) for both the preliminary and follow-up studies. See figures 1 and 2 for 
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screenshots of question in LimeSurvey. The entire questionnaire sets may be found in the full 
protocol located in appendix A. 

Use the image to identify intrusions or intrusion attempts. Click on suspicious areas of the figure. You may 
click multiple areas. To reset your selections, click "Reset". Answer any questions below. 



Locations identified as suspicious... 

(206.51666259765625.199.1999969482422).(59.516662E 


Figure 1. A screenshot of the node-link graphical representation of computer network alerts. The user is asked 
here to determine regions of the visualization that imply intrusions (True Positive [TP]) and intrusion 
attempts (False Positives [FP]) by clicking near a particular link or node. 


Use the table to identify Intrusions or possible intrusion attempts by checking the box in the associated row of data. You may tap a column heading to sort data by the 
values in that column or select a value from the drop-down box in each column to filter data. Answer any questions below. 


Suspkaou 

Date Time 

Toohame 

Protocol 

SourceEn 

SourcelP SourcePo 

DestEntrt' 

DestlP 

DestPort 

Country 

SicCC 

DstCC 

AfertMes! AlertTrafl 










r 

10/17/201 ISKKhSO 

Snort 

tcp 

US1.2 

10.234.111 52233 

MD0.6 

10.66.0.6 

52030 

MD 

US 

MD 

ETTROJAN 1090000! 

r 

10/17/201 15KK):45 

Snort 

tcp 

HD0.6 

10.66.0.6 80 

USl.2 

10.234.111 

52200 

MD 

MD 

US 

ETTROJAN PONG |3a| 

k 

10/17/201 ISKKkSO 

Snort 

tcp 

US1.2 

10.234.111 81 

MD0.6 

10.66.0.6 

52330 

MD 

US 

MD 

ETTROJAN 10900001 

r 

10/17/20115K)1:10 

Snort 

tcp 

US1.2 

10.234.111 52001 

M00.6 

10.66.0.6 

80 

MD 

US 

MD 

ETTROJAN 6ET|20|/| 

r 

10/17/201 15.-0135 

Snort 

tcp 

US1.2 

10.234.111 52001 

MD0.6 

10.66.0.6 

80 

MD 

US 

MD 

FT TROJAN 

w 

10/17/201 15:59:05 

Aggregate 

tcp 

USIJ 

10.234.111 * 

US.10 

10.250.10. 

80 

US 

US 

US 

High traffk * 

r 

10/17/201 15:20:12 

Snort 

tcp 

US1.5 

10.234.111 51119 

US.10 

10.250.10. 

80 

US 

US 

US 

ETTROJAN GET/instal 

r 

10/17/201 15:59K)0 

Aggregate 

tcp 

US1.5 

10.234.111 * 

US.10 

10.250.10. 

80 

US 

US 

US 

High traffic * 

r 

10/17/201 1532:15 

Snort 

tcp 

US.55 

10.234.111 80 

US.10 

10.250.10. 

80 

US 

US 

US 

ETTROJAN GET /instal 

[7 

10/17/20115:45.-00 

Aggregate 

tcp 

US.55 

10.234.111 • 

US.10 

10.250.10. 

80 

US 

US 

US 

High traffk * 

r 

10/17/201 15:45:50 

Snort 

tcp 

US0.2 

10.123.10C 55500 

US.10 

10.250.10. 

80 

US 

US 

US 

ETTROJAN GH /instal 

i_ i 

10/17/201 15:59:00 

Aggregate 

tcp 

US0.2 

10.123.10C • 

US.10 

10.250.10. 

80 

US 

US 

US 

High traffk * 


Checked rows are... 


3.6.10 


Figure 2. A screenshot of the tabular representation of computer network alerts. The user is asked here to 
determine which alert messages in the table imply intrusions (True Positive [TP]) and intrusion 
attempts (False Positives [FP]) by clicking the checkboxes in the ‘Suspicious’ column. 


7 
















Step 4: Review Proeess 


• The eompleted protoeol was then sent for teehnieal review by team lead, supervisor, 
external reviewers, braneh chief, division chief, and the human factors administrator. 


During the review process, there were several revisions made to the protocol. Changes included 
breaking the study up into two parts: a preliminary study to obtain initial visual display feedback 
from users, and a follow-up study to incorporate the visual displays into a cyber-network game 
scenario similar to real network analysts tasks. The approved protocol for the study with project 
number ARL 13-050 is attached as appendix A. See figure 3 for an overview of the development 
for this study. 
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Figure 3. Organization chart of an overview of the development for this study. 

3.2 Experimental Design 

Two goals for this study frame the experimental design. The first design we call Phase I includes 
three visual displays: Tabular, Parallel Coordinates, and Node-Link where we have examined 
their cognitive aspects to further identify components and representations that most effectively 
aid the CND analyst in interpreting the underlying activity in a network data sample. The second 
design we call Phase II uses a tool to capture the analyst’s cognitive reasoning process. Phase I 
investigates the various representations, and Phase II makes use of a tool designed to understand 
the process by which analysts perform their analysis. A laptop will display several figures 
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depicting network traffic using the different graphical representations. We use at least three 
visual displays: a sort-able table display (figure 4), a colored parallel coordinate display of alerts 
and normal traffic (figure 5), and a node-link display providing high-level situational awareness 
(figure 6). Instructions on how to interpret features of the visual displays provided for the 
preliminary study are provided. Their task is to examine the intrusions and intrusion attempts 
highlighted by each visual display, and to provide feedback on the effectiveness of 
communication on each representation of cyber-defense network data. 



Figure 4. Tabular Display, representation A. 



Figure 5. Parallel Coordinates Display, representation B. 
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Figure 6. Node-link Display, representation C. 

Hardware is used to conduct the experiment simultaneously with two participants. During the 
preliminary and follow-up studies, we will apply full randomization of the test subjects using the 
following possible sequences of the subjects viewing the visual displays on the hardware: 

Group 1: A^B^C 
Group 2: A^C^B 
Group 3: B^A^C 
Group 4: B^C^A 
Group 5: C^A^B 
Group 6: C^B^A 

Thus, two subjects will perform each ordering of visual displays. While not statistically 
significant, this should begin to identify any impact of the ordering on performance, which, 
itself, may aid in training of future analysts. The fabricated dataset used for the visual displays 
and game was generated using threats from ThreatExpert (37). The threats were derived from the 
“Index of Open Snort 2.9.0 Rules” (38), which is publically available. See figure 7 for an 
overview of the experiment design. 
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Figure 7. Experiment Design Overview. 


4. Future Work 


With a complete and approved protoeol, we ean now begin the preliminary and follow-up studies 
for the next fiscal year. The goal is to conduct the study with both the expert analysts and student 
users, eollect the data, and analyze (aeeuraey, error rate, time, and quantitative questionnaires) 
the results. We plan to submit a teehnieal report of our findings and to publish a paper for a 
conference or journal. 
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Engineering Visualization Research Laboratory (EVRL) 

Schaefer Engineering Building, Room 112 
Morgan State University, Baltimore, MD 

Abstract 

The goal of security visualization is to help analysts increase the safety and soundness of our 
digital infrastructures by providing effective tools and workstations {16). Analysts have a 
difficult task characterized by the need to integrate technical knowledge with contextual 
knowledge under severe constraints. In our case, we plan to turn visual overload into an 
opportunity by enhancing the visual displays used by analysts into a more effective tool. 
Traditionally, analysts are used to working with tabular displays for conducting their tasks. Other 
types of displays such as a graphical display may provide more insight into big data inter¬ 
relationships, patterns, and finding areas of interest. In response, we have designed an 
experiment using cyber-network data to test the effectiveness for communicating suspicious 
activity on a computer network through visual displays. In the study, participants act as analysts 
and their job is to identify as many as possible of the intrusion attacks and intrusion attack 
attempts on the tabular and graphical displays provided. The design variables will be several 
distinct graphical layouts. The response variables are true positive and false positive rates of 
event identification, the time required for event identification, and a qualitative questionnaire. 
Results will help us understand which of the visual layouts is most effective for predicting cyber 
attacks. This will benefit network security analysts who defend the nation’s networks. 

Location of Research 

We will conduct the research at Adelphi Laboratory Center (ALC), Aberdeen Proving Ground 
(APG), and Morgan State University (MSU). 

Data Collection Dates 

The data collection dates will take place 1 September 2013 through 1 October 2014. 

Study Sponsor 

U.S. Army Research Laboratory Computational and Information Sciences Directorate (ARL- 
CISD) 

Research Background 

Millions of data features can quantify the structure of complex cyber networks. However, 
information overload is a persistent problem existing in graphical layout techniques. In a graph, 
nodes represent objects under analysis and links represent the relationship between these 
elements. The drawing of nodes and their edges onto a two-dimensional surface is a difficult 
problem with no satisfying solution. Challenges arise in the representation of graphs visually due 
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to the complexity of the graphs and the difficulty in removing occlusion in 2D projections while 
representing the overarching ontology. 

Tables work best for representing data when the presentation is used to look up or compare 
individual values, when precise values are required, and when the values involve multiple units 
of measure {17). Graphical representations, work best when the data presentation is used either to 
communicate a message that is contained in the shape of the data or to reveal the relationship 
among many values. Hence, graphs and tables are the two primary means to structure and 
communicate quantitative information. Surprisingly, not much work has been done in the cyber 
security domain to validate or disprove the effectiveness of either display type being used in the 
analysis of cyber security tasks. Goodall (79) is one of few whose work focused on comparing 
user performance by using two different tools designed to analyze captured network packet data. 
Traffic Network Visualization tool (TNV) was the visualization-based display tool used. It was 
compared to a textual-tabular-based display tool. In (79), TNV proved increased accuracy for 
well-defined tasks. They also mention a clear preference from their expert participants for the 
visual interface. While other graphical visualization tools have been developed and prove useful 
such as Koike and Ohno’s SnortView {20) that used simple geometric shapes to indicate protocol 
and severity in two-dimensional grid relating source IP address to destination IP address and 
time, Goodall remains the only known approach comparing tabular and graphical displays 
validated by a user study. Our approach to enhancing analyst’s needs for visual displays differs 
from Goodall in two ways, the tools used and the focus of what is being visualized. We initially 
use the MATLAB tool to house a tabular display that will be compared to several graphical 
displays and we focus on visualizing network traffic monitored by analysts rather than the 
correlated IDS output in (79). Our results will validate improved accuracy and the desire for 
visual displays that are more effective. Our study helps in creating a sound voice and reference 
for the cyber security domain concerning this matter. We agree with the authors (27) about the 
importance of grounding cyber security visualizations through user studies. 

Ongoing research continues to look for new ways to provide decision-making opportunities that 
improve the effectiveness of cyber-security network analysts’ activities. Effective visualization 
techniques can identify predictive features and reduce the dimensionality of both data and model 
while identifying relevant patterns (d). We aim to understand the underlying characteristics of 
effective cyber security monitoring such that we can minimize the information displayed to 
analysts. The ultimate goal for an effective display is to improve task performance enhancing 
situational awareness accuracy or decrease cognitive load. Contributing factors for cognitive load 
include perception, problem solving, and multi-tasking. We want to identify the salient features 
that analysts respond to best for each graphical layout of the visualization tool’s environment. 

The relevance of the study is to help us better focus on aspects within the visual representation 
that may require cognition. Our informal hypothesis is that there will be better knowledge of 
analyst response to visual stimuli that will allow the generation of visual representations to 
maximize saliency of features of interest for network analysts. 


17 



Previous work on visualization for cyber security has focused on data analysis, event analysis, 
event identification, and situational awareness (7). These studies have proposed visual 
alternatives but none have been tested on significantly large network data sample sizes, equitable 
data simulations, or using expert network analysts. An example of such visual alternatives is 
from Kosara, et al.’s semantic depth of field, in which renderings strive to induce perceptual 
changes in the user (4). Tory and Moller (5) offer a thorough discussion of human factors in user 
study methods, and visualization design. 

Evaluating visualization techniques can be a difficult task. The primary approaches include 
subjective feedback from domain experts and quantitative user studies. We will employ 
quantitative user studies as our primary approach in this study. There are several methods for 
measuring user response during visual user studies. This includes direct user manipulation. 
Electroencephalography (EEG) and Eunctional Magnetic Resonance Imaging (EMRI). We will 
employ direct user manipulation in which users directly respond to display elements while their 
time and performance are measured. EEG, a process of passively recording brain activity, is an 
alternative method for quantitatively evaluating visualization techniques. The measurements 
collected by EEG determine the amount of burden placed on an individual’s cognitive resources. 
Anderson et al. (8) used this method for analysis of their visualization technique. Another 
method is EMRI, which is the process of detecting changes in blood oxygenation and flow that 
occur in response to neural activity (JO). An active brain area consumes more oxygen and in 
response to the demand blood flow increases to that area. This method produces activation maps 
that show which parts of the brain are involved in a particular brain activity (JO). Unfortunately, 
there are some disadvantages to using EMRI. It is expensive, clear images are only captured if 
the person being scanned remains completely still, and researchers are still uncertain of how 
EMRI really works. The disadvantages of using EEG are that it provides a view of overall brain 
activity, which is not specific to different areas of the brain and it requires attachment of 
electrodes to the subjects. We leave measuring cognitive load with EEG or EMRI to future 
studies. 

Research Objective 

This empirical study will examine the cognitive aspects of visual displays with the goal of 
identifying components and representations that most effectively aid the computer network 
analyst in interpreting the underlying activity in a network data sample. An additional objective 
is to capture the analyst’s cognitive reasoning process via analysts recording their sequence of 
thoughts while conducting network defense tasks. The understanding obtained from the results in 
this study will allow for the generation of visual representations that maximize saliency of 
features of interest for network analysts and aid in building the foundation for science and theory 
of network-based intrusion detection. A preliminary study will gather preliminary results via 
subjects’ response to generated visual representations of cyber-network data presented in capable 
environments such as the MATEAB tool. We also plan to conduct a follow-up study using a 
cyber-network attack game scenario to support our study’s objectives. The game will identify 
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characteristics from the visual displays that are more effective for cyber analysis by using the 
results of student subjects versus expert subjects in a cyber-network game scenario. 

Instrumentation and Facilities 

Equipment 

• At least two Laptops 

• Installation/use of the visualization software (i.e., MATLAB etc.) on all hardware 

• Installation/use of PSU trace tool on all laptops 

• Installation/use of game software on all laptops (for follow-up study only) 

• One projection machine 

Safety Releases for Equipment or Apparatus 
No safety releases are required. 

Eacility 

We plan to conduct the studies at locations convenient to both network analysts and students. In 
particular, we plan to use the System Assessment and Usability Laboratory (SAUL) on the 
second floor of building 459 at APG, MD for subjects at that location and use room 2F014 in 
building 204 for subjects at the ALC location. There are several vacant rooms on the 2F00 
hallway available for this study. SAUL and these rooms are good fits for this study because their 
primary function is to execute studies for software user interfaces. 

Standard Operating Procedures for Courses or Eacilities 

There is currently no SOP on file for these facilities. 

Materials, Tests, Tasks, and Stimuli 

There are three questionnaires prepared for this study. Participants will sign the consent form and 
then take the surveys. The first is a “Demographic Questionnaire” which asks the subjects to 
provide background information and their level of experience related to the domain of the study 
(appendix B). The “Subjective Questionnaire” allows the subject to rate their experience with the 
game scenario manipulation of the graphical layout (appendix C). Last, a “Survey 
Questionnaire” asks the subject to rate their overall experience, identify their preferences 
associated with the different visual displays, and assess their use of the visualization tool 
(appendix D). 

Tasks and Stimuli 

A laptop will display several figures depicting network traffic using different graphical layout 
modes such as the one depicted in (appendix A). The participants will be given instructions on 
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how to interpret features of the visual displays provided for the preliminary study. Their task is 
to examine the intrusions and intrusion attempts highlighted by each visual display and to 
provide feedback on the effectiveness of communication on each representation of cyber-defense 
network data. We will use the hardware to conduct simultaneously the experiment with two 
participants. 

During the preliminary study, we will make use of MATLAB, a high-level language and 
interactive environment for numerical computation, visualization, and programming to 
implement the designated visual display paradigms. MATLAB provides tools that enable gaining 
insight into data. Documents of explored analysis can be created and shared as reports or 
published MATLAB code (22). MATLAB allows access to data from files, other applications, 
databases, and external devices that may be read in via popular file formats such as Microsoft 
Excel; text or binary files; image, sound, and video files; and scientific files such as netCDF and 
HDF (22). The tool’s ability to perform exploratory data analysis to uncover trends, test 
assumptions, and build descriptive models is one of the main reasons we selected it for use in 
this study. 

During the follow-up study, we plan to use a program developed for ARE called CyFall. We will 
tell the participants that they are playing a game where they will be acting as a real cyber analyst. 
The participants’ goal within the game is to try to detect and identify all of the intrusions and 
intrusion attempts on the network, as presented by the visual display. The participants will use 
the same visual displays from the preliminary study to identify and label as many of the 
correlated pieces of evidence that exist for each incident. 

Subjects 

The participant population will consist of analysts from ARE CISD located at AFC and APG as 
well as students from Morgan State University. The analysts are the individuals who either 
currently or in the past used related visual displays for the performance of their daily activities. 
This group will consist of eight to twelve participants. Subjects can only participate in this study 
if they are eighteen or older. For this preliminary study and the follow-up study, sixteen to 
twenty-four participants are sufficient. Subjects will be recruited by direct solicitation via 
personal and e-mail communication. There is no supervisory pressure to participate in this 
human research study. The subject is free to leave the study at any time. The follow up study will 
also be conducted at Morgan State University to compare student subjects with the expert 
analysts (ARE subjects). 

Sample Size Justification 

The preliminary and follow-up studies are initial usability tests. We will use the results to guide 
the design of actual visual displays to be used by network analysts. Therefore, only a few users 
are required. Sixteen to twenty-four participants will be recruited for this study. This number of 
participants is more than sufficient to gather qualitative feedback. 
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Compensation 

Participation in the study is voluntary and there is no compensation provided for the subject’s 
time or input. However, their contributions and results of the study may improve the quality of 
visual displays for ARL analysts, allowing them to identify features of interest more efficiently 
and effectively. 

Subject Recruitment 

Subject recruitment will be done by direct solicitation via word of mouth and email. We will 
specifically target analysts from the Sustaining Base Network Assurance Branch (SBNAB) 
within ARL CISD. Initial contact will be made to the SBNAB branch chief and he will notify us 
of arrangements on how to proceed with recruiting analysts from SBNAB. See Appendix J for 
the initial ARL recruitment email. 

Experimental Design 

As noted in the “Research Objective” section, the study is divided into two experimental efforts; 
a preliminary study intended to gain basic insights into the different representations and a more 
comprehensive follow-up study that uses a cyber-defense network game scenario. Each of these 
studies is divided into two phases: Phase I investigates the various representations and Phase II 
makes use of a tool designed to understand better the process by which analysts perform their 
analysis. Each of these components is described in this section. 

Phase I: 

Eor Phase I, both the preliminary and follow-up studies use post-experiment surveys in 
conjunction with timing and task-related data to form a foundation for additional statistical 
analysis. The following types of visual representations are used: 

• A tabular sort-able display, see figure A-1 
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Figure A-1. Tabular Display, representation A. 

• A colored parallel eoordinate representation of alerts and normal traffie with a data 
inspector pane, see figure A-2 
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Figure A-2. Parallel Coordinates Display, representation B. 


• A node-edge representation providing high-level situational awareness, see figure A-3 



Figure A-3. Node-link Display, representation C. 
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Thus, there are at least three different ways to represent the same set of information. Design 
variables for both the preliminary and follow up studies are at least three graphical layouts. The 
response variables are true positive and false positive rates of event identification, the time 
required for event identification, and the qualitative questionnaires. 

The Tabular Display provides data representing the exact attribution of the entire system 
typically presented in Microsoft Excel. An incident here is described as known bad senders, 
suspicious use of particular ports, and known patterns in the data packets. A list of alerts from a 
one-hour period is provided where the analyst can search for threats. Successful identification of 
a threat is considered a true positive (TP). Of course, it is a given that there will be some number 
of false alarm alerts we call false positives (FP). The analyst has a major task of differentiating 
the real threats from the false alarms. Figure A-4 for an example of what these alerts look like on 
the Tabular Display. For the follow-up study, the Tabular Display uses forensic techniques to 
collect and group evidence into what we call the victim system, which is the screen, on the 
display. 
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Figure A-4. Alerts Example in a Tabular Display. 

The example alerts in figure A-4 are typically displayed in a tabular format, figure A-1, and 
analysts are very good at correlating the data to identify events of interest. In addition to the 
tabular format, we will also examine at least two more cognitively oriented visual displays, 
figures A-2 and A-3. During the preliminary and follow-up studies, we will apply full 
randomization of the test subjects, which amounts to the following possible sequences of the 
subjects using the visual displays: 

Group 1: A^B^C 
Group 2: A^C^B 
Group 3: B^A^C 
Group 4: B^C^A 
Group 5: C^A^B 
Group 6: C^B^A 
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Thus, two subjects will perform each ordering of visual displays. While not statistically 
significant, this should begin to identify any impact of the ordering on performance, which itself 
may aid in training of future analysts. 

The Parallel Coordinates Display was originally generated by a tool called GUESS, an 
exploratory data analysis and visualization tool for graphs and networks (9). GUESS contains a 
domain-specific embedded language called Gython, an extension of Jython. Jython is a Java 
based language derived from Python. Gython supports the operators and syntax necessary for 
working on graph structures in an intuitive manner. The tool also offers a visualization front end 
that supports the export of static images and dynamic movies. We selected GUESS because 
Army Research Eaboratory researchers preferred its ease of use; alternative development 
environments may be used but the displays and interactions will remain essentially similar to 
what is described. We chose this tool to represent interactions within a network. With the tool, 
we show a single connection from one system to another as a solid directed line. Earge hashes 
along the directed line represent users who have multiple connections to more than one system. 

A single hash mark represents each user. With this information, we can measure activity between 
systems and monitor behavior patterns. We use red to highlight unusual or unexpected activity 
{ 11 ). Eigure A-5 for the visual key of these directed line representations. 

(a) A solid directed line represents a 
connection from one system to another 

(b) A long dashed directed line represents users 
with multiple connections 

(c) A short dashed directed line represents each 
user 

(d) A solid directed line with many arrows 
represents a Network Eile System (NES) 
access 

(e) A double line with an arrow represents an 
initial port connection 

(f) A solid red directed line represents unusual 
or unexpected activity 

Figure A-5. (a) Line Visualizations Used for Parallel Coordinate Display, (b) Meanings of Line 
Visualizations Used (77). 

Eor the Node-Eink Display, glyph-based visual representations are created as visual attributes to 
portray connections that could exist within any system. These parameters include but are not 
limited to number of users, system load, status, and unusual or unexpected activity { 11 ). Eor the 
preliminary study, we introduce these glyph representations and ask the participants for their 
response to effectiveness in communicating network parameters of a system’s data. The result is 
a display of visual attributes that are easily interpretable for their actual meaning. In the follow¬ 
up study, the visual attributes are designed in a cyber-defense network game scenario in 
conjunction with the database parameters in such a way that the correlation is appropriate and the 


a) -> 

b) -> 

. -> 

e) -X 
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relationship is comprehensible to the analyst. Figure A-6 for the visual key of the glyphs 
representations and meaning. 



(a) A basic glyph that represents the initial 
connection to a system (note the double lines) 

(b) A basic glyph that represents the resulting 
connection to a system after authentication 
(notice the single line) 

(c) A basic glyph that represents the number 
of users (solid lines drawn from the circle 
outward) to the load (the circle) 


Figure A-6. (a) Line Visualizations Used for Node-Link Display, (b) Meanings of Line Visualizations Used (77). 
Preliminary Study (no game scenario) 

Participants will detect the intrusions and possible intrusions on the visual representations. 
Subjects will examine the highlighted features within each visual representation to determine 
which alerts are intrusions or possible intrusions. They will then provide feedback on the 
effectiveness of the communication on each visual representation of cyber-defense network data. 
In this study, an intrusion is defined as the ability to compromise a computer system by breaking 
the security of the system or by causing it to go into an insecure state. We assess a possible 
intrusion by the identification of events that occur close together in time. 

We use MATLAB as the environment to display the visual representations, see figure A-7. 


25 



• MATHLAB is good for data 
analysis and visualization 

• Able to acquire data from files, 
other applications, databases, and 
external devices 

• Able to filter, manage, and 
preprocess data 

• Provides built-in 2D and 3D 
plotting and volume visualization 
functions 

• Documenting and sharing results 
are possible via plot or reports 

• Reports can be published in a 
variety of formats, such as HTML, 
PDF, Word, or LaTeX 

Figure A-7. (a) MATLAB shot of plot library being used, (b) MATHLAB highlights. 

Follow-Up Study (with the same scenario) 

Participants will detect the intrusions and possible intrusions on the simulated network via a 
game scenario by doing the following: 

1. Correlate the different alerts by foreign IP address. A number of different types of alerts 
coming from the same source are extra suspicious. Try to gather, by sorting, all the traffic 
to and from the same IP address. A secondary sort should be on the local IP - separate the 
messaging with different local addresses. The game also has some additional visual graphs 
and pictures showing traffic volume, separated by foreign IP address. 

2. Look for a malware or Trojan name in the “Alert Message Emitted” text. If the alert 
identifies a particular piece of malware, it is more likely to be a real threat. It would still 
have to be correlated with other traffic to make it more certain. 

3. The foreign country in the source or destination field can usually be calculated based upon 
the IP address. This is only an indicator, because there is legitimate traffic from unfriendly 
countries and threatening traffic that appears to come from friendly countries. 

We designed a representative dataset that contains a number of ‘alert’ records displayed for an 
equivalent large site. The fabricated set will contain 500 ‘alert’ records because typically an 
analyst sees 500 alerts during an hour. However, less than one percent of these alerts actually 
correlate to an incident or interesting feature of traffic. The game scenario uses the visual 
displays to illustrate the designed dataset differently. The participants are given instructions on 
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how to play the game, which incorporates the above mentions tasks. To win the game, a 
participant must correctly identify all of the intrusions and intrusion attempts on each level (a 
level represents a visual display). Figure A-8 that shows screenshots of the CyFall tool 
developed for the game scenario. 
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Figure A-8. (a) This is a screenshot of the “Introduction Overlay”. It introduces the subject to the mission, 
provides instructions on identifying threats, and highlights the features/functions of the game. 

(b) This is a screenshot of the “Exercise Overlay”. Here the subjects will be able to view, 
explore, and look deeper into the dataset via the particular visualization display. It is here after 
exploration that the subject determines and identifies the network threat, (c) This is a screenshot 
of the “Results Overlay”. A running log is kept in the background to keep track of each subject's 
performance. A module contains all three overlays and repeats three times for the three different 
visual displays. The subject's performance is displayed at the end of each module and once more 
at the end of the entire session for their overall time and accuracy performance. 
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Phase II: 


A second goal of this investigation is to begin to understand the process by which analysts 
perform their analysis. This cognitive process is of great interest since algorithmic approaches 
have, to date, been unable to duplicate evenly this process remotely. Understanding more about 
this cognitive process will enable development of tools designed to aid the analysis process as 
well as the development of algorithms to reproduce said process, particularly in the case of 
known threats. To this end, a second phase of the protocol will examine an experience-aided 
reasoning support system developed at Penn State University (PSU) under an Army Research 
Office (ARO) MURI, see figure A-9. They have designed this tool to both support analysts in the 
development and evaluation of a hypothesis as well as record the analyst’s process of evaluating 
and rejecting or accepting hypothesis. We will make no association between the recorded 
processes and the analysts name or identifying characteristics. Again, performance metrics with 
and without the aid of this tool will be generated for quantitative analysis. In addition to the tool, 
it is also feasible to have analysts dictate their process, as implied earlier. There is a goal of 
analyzing the cognitive process tool. 


1) Data Monitoring 


2 

NetwDiit Comedian | IDS Log | Packet Dump | Web Senret Log | Auth 

jog 1 AnU-viue (Report | Wjlnerabih j 

T 

S 

Nodei 

1 Pot 

Stale 


VafUon 


_ 



k l21Acp 

open 

ftp 

Microsoft ftpd 




DNS 130 203 50.2 

l25acp 


smto 

Mcrosoft ESMTP 6.0.3790.3959 





[soicp 

open 

http 

iMcrosoft IIS Webserver 6.0 


< 


PCI 130.203.158.101 

iioacp 

open 

pops 

MS Exchwige2003pop3d 6.5.763S.1 










I Eight monitoring 
I data sets 


(2) Hypotheses Navigation 



(3) Experience Guidance 


Case Study: 
Two multistep 
attack chains 


Figure A-9. Experience-aided reasoning support system overview. 

Subjects here have a chance to practice analyzing the data provided by the system. We introduce 
participants to IDS alerts, network configuration, vulnerability reports, data dumps, port scanner 
reports, and system logs. The subjects are encouraged to speak out aloud during their thinking 
process of coming up with and finalizing their hypothesis. 

Thus for Phase II of the study, we are specifically asking the participants to (for both preliminary 
and follow-up studies): 
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1. Conduct analysis on the provided network data. 

2. Create hypotheses (notations of thoughts in making decisions within the presented network 

data). 

3. Participants are encouraged to strike original thoughts and make new ones as they would in 

a natural cognitive thinking process. 

The participants’ personal performance in this study is not the focus of this research. Instead, 
their performance helps us to generate better visual representations that maximize saliency of 
features of interest for network analysts’ intrusion detection tasks. In addition, the results aid in 
building the foundation for the science and theory of intrusion detection. Participants will have a 
maximum of three hours to complete the tasks in a sitting. 

Procedure 

1. Preliminary Study 

Step 1. We will begin the study with a welcome followed by an introduction of the 
investigators. 

Step 2. Investigators will then brief the participants on the study and will obtain informed 
consent. Participants of this study will be given a random anonymous 
identification number to protect their personal information and identity. They will 
be asked to complete a background and demographics questionnaire. 

Step 3. The investigators will explain each visual display used and their specific ways of 
representing a network system’s attributes. 

Step 4. The investigators will ask the subjects to record their thought process in making 
their decisions. The PSU tool collects these notations as hypotheses (thoughts) 
and creates a tree that traces a subject’s thoughts throughout the experiment. 

Step 5. The investigators will then describe the tools and explain how the participants will 
use them. 

Step 6. The investigators will conduct a run-through or demo if you will of the 

participants tasks. This will serve as practice for the participants. We demonstrate 
how to create a new hypothesis by clicking the mouse on the trace submission 
button. 

Step 7. The investigators will lead a session to entertain questions that the participants 
might have concerning their tasks or any other aspects of the study. 

Step 8. Participants will conduct the experiments for Phase I (visual representations only) 
and Phase II. 
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Step 9. Participants will complete their post-task questionnaires and provide the 
investigators with any final remarks or comments. 

Step 10. Investigators will lead a debrief session and provide the participants with a copy 
of the signed consent form. 

Participants will have a maximum of three hours to complete the tasks in one sitting. See 
Appendix B for the Pre-Task Questionnaires and Appendix C for General Background 
Information ask of the participants. See Appendix D through Appendix H for the Post-Task 
Questionnaires. 

2. Follow-Up Study 

Step 1. We will begin the study with a welcome followed by an introduction of the 
investigators. 

Step 2. Investigators will then brief the participants on the study and will obtain informed 
consent. Participants of this study will be given a random anonymous 
identification number to protect their personal information and identity. They will 
be asked to complete a background and demographics questionnaire. 

Step 3. The investigators will describe the cyber-network game scenario and the 
participant’s associated tasks as a cyber-security analyst. 

Step 4. The investigators will ask the subjects to record their thought process in their 
making their decisions. The PSU tool collects these notations as hypotheses 
(thoughts) and creates a tree that traces a subject’s thoughts throughout the 
experiment. 

Step 5. The investigators will then describe the tools and explain how the participants will 
use them. 

Step 6. The investigators will conduct a run-through or demo if you will of the 
participants tasks. This will serve as practice for the participants. 

Step 7. The investigators will lead a session to entertain questions that the participants 
might have concerning their tasks or any other aspects of the study. 

Step 8. Participants will conduct the experiments for Phase I (game scenario) and Phase 
II. 

Step 9. Participants will complete their post-task questionnaires and provide the 
investigators with any final remarks or comments. 

Step 10. Investigators will lead a debrief session and provide the participants with a copy 
of the signed consent form. 
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Participants will have a maximum of three hours to complete the tasks in one sitting. See 
appendix B for the Pre-Task Questionnaires and appendix C for General Background 
Information ask of the participants. See appendix D through appendix H for the Post-Task 
Questionnaires. 

Data Analysis 

The performance of the participants will be monitored throughout the entire experiment: 
Preliminary Study 

We will measure participants’ ability to correctly identify suspicious activity (intrusion attempts) 
and cyber attacks (TP-true positive match for a network threat) using each of the different 
displays and compare performance among the representations. We also use questionnaires to 
measure the subject’s subjective perception of representations. 

Follow-up Study 

The time for each participant to complete the entire scenario will be recorded as “Total Time”. 
We will perform statistical analysis by measuring the effectiveness of the comparisons between 
the input visual displays and by identifying detections versus Total Time. We use log recording 
to collect the number of intrusion attempts and intrusion attacks correctly identified by the 
subject. The second metric is the computation of error rate for a strict definition of True Positive 
as a right answer. The effects of the different visual display types on error rate will be compared. 
The third metric are scores from the questionnaires themselves to measure the subject’s 
performance. In addition, noted for follow up with the participants, are observable difficulties 
with the displays or extreme lag time of no action. These results should identity features from the 
visual graphical displays that are effective for cyber security. 

Risks 

The study involves minimal risk and minimal discomfort to the participants; the analysts in 
particular are regularly required to work twelve-hour shifts in front of a computer as part of their 
assigned duties. The likelihood of any physical, mental, or emotional harm is negligible. There 
will be no psychologically or physically exhausting work required. The investigators will 
monitor the safety of the participants in this study however; we cannot eliminate all discomforts 
that may occur. The following are possible discomforts for this study: 

1. Subjects may experience eyestrain in a dimmed light setting during this study. 

2. Subjects may experience unexpected discomforts such as sitting discomforts in this study. 
There is a risk of back pain, leg pain, arm pain, or any other associated pain with sitting for 
an extended period. 
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Benefits 


There is not an immediate benefit to the participants. However, their contributions and results of 
the study may improve the quality of the visual display for ARL analysts, allowing them to 
identify features of interest more efficiently and effectively. 

Confidentiality 

The participants’ personal information remains confidential. The study requires obtaining basic 
information from participants and no personal information besides a name and signature for the 
consent form is required. This study uses the participants’ responses, performance, and 
demographic information related to the study in the publication of the research. However, we 
provide a random anonymous identification number to protect their identity and results for 
publication. Participants are neither photographed nor videotaped. We will use audio tapes to 
record their interview responses ensuring clarity and accuracy of their responses. Researchers 
will review the audio recordings and ensure that no personally identifying information or other 
sensitive details will be released to the public. Of course, a participant is free to retract their 
statements during the interview session. 
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Appendix A: Consent Form 


Site of Research: Building 459, Room 202 System Assessment and Usability 
Laboratory (SAUL), Aberdeen Proving Ground, MD 

Research Participant Consent Form 
Army Research Laboratory 


Project Title: 


Evaluation of the Presentation of Network Data via Visualization 
Tools for Network Analysts 


Sponsor: 


Department of Defense 


Co-Principal Investigator: Renee E. Etoty, Adelphi Eaboratory Center, MD, 301-394-1835, 

renee.e.etotv.civ@mail.mil 


Co-Principal Investigator: Robert E. Erbacher, Adelphi Eaboratory Center, MD, 301-394- 

1674, robert.f.erbacher.civ @mail.mil 


Associate Investigator: Christopher Garneau, Aberdeen Proving Ground, MD, 410-278- 

5814, Christopher.].garneau.civ@mail.mil 


Date: 


03 April 2013 


We are asking you to join a research study. This consent form explains the research study and 
your part in it. Please read this form carefully before you decide to take part. You can take as 
much time as you need. Please ask questions at any time about anything you do not understand. 
You are a volunteer. If you join the study, you can change your mind later. You can decide not to 
take part right now or you can quit at any time later on. 

Why is this research being done? 

We invite you to participate in a study deigned to assess visual layouts for cyber-security 
network analysts on representative network activity; in essence, we will use computer graphical 
displays to represent computer network activity that the network analysts currently view in a 
tabular format. This study will examine the cognitive aspects of visual displays with the goal of 
identifying representations and components of representations that most effectively aid network 
analyst in interpreting the underlying activity in a network data sample. The Army Research 
Eaboratory (ARE) - Computational Sciences, Information Directorate (CISD), and Human 
Research Engineering Directorate (HRED), are conducting the study. 

What will happen if you join this study? 

As a participant of this study, we will give you a random anonymous identification number to 
protect your personal information and identity. We will ask you to complete a background 
experience and demographics questionnaire. An investigator will describe the tasks for the 
Preliminary Study or Eollow-Up Study. We describe the specific tools and tasks below. You will 
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have a maximum of three hours to complete the tasks in a sitting. There will be a fifteen-minute 
session after a short demonstration-training period to address your questions, comments, and 
concerns. Upon completion of the tasks, we will ask you to execute a post-task questionnaire that 
will reflect your comments about the overall study experience and your impression of the visual 
displays used. 

Particularly for the Preliminary Study, specifically, we ask you to detect the intrusions and 
possible intrusions on the visual representations provided. You will examine the highlighted 
features within each visual representation to determine which alerts are intrusions or intrusion 
attempts. You will then provide feedback on the effectiveness of the communication on each 
visual representation of cyber-defense network data. In this study, an intrusion is defined as the 
ability to compromise a computer system by breaking the security of the system or by causing it 
to go into an insecure state. We assess a possible intrusion by the identification of events that 
occur close together in time. 

For the Follow-Up Study, we ask you to play a cyber-network game scenario that incorporates a 
pattern matching behavior representative of a typical cyber-security analysis session. Your goal 
for the game is to identify all the intrusions and intrusion attempts made by the cyber attacker. 
This technique is coupled with a visual display that aids an analyst in performing their tasks. The 
visual task scenarios of the game will compare analyst effectiveness across at least three 
constructed visual layouts. 

We provide you with the tools used to carry out your tasks for both studies. These tools consist 
of at least three types of displays showing network activity of interest to network analysts. The 
first display is a sort-able table. The second display is a colored parallel coordinate 
representation of alerts and normal traffic with a data inspector pane. The third display is a 
“node-edge” representation. A fabricated set of data that contains 500 ‘alert’ records will be 
presented in the study an analyst typically sees about 500 alerts during an hour. You will be able 
to manipulate this dataset on each visual display to help better identify intrusions on the 
simulated network in the game. 

The second phase of the experiment of both studies requires you to practice developing a 
hypothesis (theory) of the status of the given system. We give you the opportunity to conduct an 
evaluation of your developed hypothesis. We will record your process of evaluating and rejecting 
or accepting the hypothesis using log files. 

Your personal performance in this study is not the focus of this research. Instead, your 
performance helps us to generate better visual representations that maximize saliency of features 
of interest for network analysts’ intrusion detection tasks. Also, note that we will record your 
performance and we will in no way disclose this information to your respective communities nor 
publish any identifying information that is traceable back to you. 

How much time will the study take? 

Your participation in this study will take up to a maximum of three hours for one sitting. 
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What are the risks or discomforts of the study? 

The likelihood of any physical, mental, or emotional harm is remote. There will be no 
psychologically or physically exhausting work required. The investigators will monitor your 
safety however, we cannot eliminate all discomforts that may occur. The following are possible 
discomforts for this study: 

1. You may experience eyestrain in a dimmed light setting during this study. 

2. You may experience discomforts due to sitting for an extended period during this study, (e.g., 
there is a risk of back pain, leg pain, arm pain, or any other associated pain with sitting for an 
extended period). 

Are there benefits to being in the study? 

To you as the participant, there is no immediate benefit for participating in this study. Your 
participation as a student subject or an expert subject allows us to use your results and feedback 
to improve the generation of visual representations that maximize saliency of features of interest 
for network analysts. This leads to better quality of the visual displays for cyber-security 
analysts, allowing them to identify features of interest more efficiently and effectively. 

Will you be paid if you join this study? 

You will receive no payment for taking part in this study. 

How will your privacy be protected? 

We will keep your personal information confidential. Your personal information will be stored 
and secured in a locked and password protected computer at our study site. After transfer of your 
personal information to our secured computer, we will shred the paper copies containing your 
personal information. In addition, we provide a random anonymous identification number to 
protect your identity and your results. Publication of the results of this study in a journal or 
technical report or presentation at a meeting will not reveal personally identifiable information. 
We will neither photograph nor videotape you. The investigators will further protect your 
personal information from disclosure to individuals not connected with this study. However, we 
cannot guarantee complete confidentiality because law permits officials of the U. S. Army 
Human Research Protections Office and the Army Research Laboratory’s Institutional Review 
Board to inspect the records obtained in this study to insure compliance with laws and 
regulations covering experiments using human subjects. The principal investigator will retain 
this consent form for a minimum of three years. 

Indicate below if we have your permission to audio record you during the experimental session. 
We will use audio recordings to record your interview responses ensuring clarity and accuracy of 
your responses. Please indicate below if you will agree to allow us to record you. You can still 
participate in this study if you prefer not to be audio recorded. 

I give consent to be audio taped during this study: _Yes_No please initial:_ 
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Where can I get more information? 


You have the right to obtain answers to any questions you might have about this research both 
while you take part in the study and after you leave the research site. Please contact anyone listed 
at the top of the first page of this consent form for more information about this study. You may 
also contact the chairperson of the Human Research & Engineering Directorate, Institution 
Review Board, at (410) 278-5992 with questions, complaints, or concerns about this research, or 
if you feel this study has harmed you. The chairperson can also answer questions about your 
rights as a research participant. You may also call the chairperson’s number if you cannot reach 
the research team or wish to talk to someone who is not a member of the research team. 


Voluntary Participation 

Your decision to be in this research is voluntary. You can stop at any time. You do not have to 
answer any questions you do not want to answer. Refusal to take part in or withdrawal from this 
study will involve no penalty or loss of benefits you would receive by staying in it. 

Military personnel cannot be punished under the Uniform Code of Military Justice for choosing 
not to take part in or withdrawing from this study, and cannot receive administrative sanctions 
for choosing not to participate. Civilian or contractor personnel cannot receive administrative 
sanctions for choosing not to participate in or withdrawing from this study. Once we have 
answered your questions about the study, and if you want to continue your participation in this 
study, please sign below. 

We Will Give You A Copy Of This Consent Form 


Signature of Participant 


Printed Name 


Date 


Signature of Person Obtaining Consent Printed Name 


Date 
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Appendix B: Pre-Task Questionnaire 


Demographic Information 


1) 

What is your gender? 


a. 

Male 


b. 

Female 

2) 

What is your race? 


a. 

American Indian or Alaska Native 


b. 

Asian 


c. 

Black or African American 


d. 

Native Hawaiian or Other Pacific Islander 


e. 

Other 


f. 

White 


g- 

Prefer not to say 

3) 

What is your age? 


a. 

18-25 years old 


b. 

26-35 years old 


c. 

36-45 years old 


d. 

46-55 years old 


e. 

56-65years old 


f. 

66-75years old 


g- 

76 years or older 

4) 

What is the highest level of education you have completed? 


a. 

Elementary school only 


b. 

Some high school, but did not finish 


c. 

Completed high school 


d. 

Some college, but did not finish 


e. 

Two-year college degree / A.A / A.S. 


f. 

Four-year college degree / B.A. / B.S. 


g- 

Some graduate work 


h. 

Completed Masters or professional degree 


i. 

Advanced Graduate work /Ph.D. 

5) 

What is 

; your work title? 

6) 

What is your current department? 
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7) 


Do you have any vision impairments or poor vision (after correction)? 

a. Yes 

b. No 

c. If yes, please explain. 


8) Do you have any other disabilities? 

a. Yes 

b. No 

c. If yes, please explain. 
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Appendix C: General Background Information 


1) Do you use computers (PC’s, MAC, iPad, iPhone, Android phone, tablets, etc.)? 

a. Yes 

b. No 

2) How often on a daily basis, do you use computers? 

a. 1-5 hrs 

b. 5-10 hrs 

c. 10-15 hrs 

d. 15-20 hrs 

e. More than 20 hrs 

3) How comfortable do you feel using a computer? 

a. Very comfortable 

b. Somewhat comfortable 

c. Somewhat uncomfortable 

d. Very uncomfortable 

4) Have you ever written a software program or a mini computer code? 

a. Yes 

b. No 

5) Have you ever configured a Linux computer? 

a. Yes 

b. No 

6) What is a shell? 


7) Have you ever worked as a network analyst or have any network analysis 
experience? If so, state where and when. 

a. Yes 

b. No 

c. State your experience. 
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If an analyst, answer questions #8 and #9. 

8) How many years have you been a cyber analyst? 

a. Less than 1 year 

b. 1 to 3 years 

c. 3 to 5 years 

d. 5 to 10 years 

e. More than 10 years 

g. Never 

9) Which of the following activities do you most frequently perform in your work? 
Check all that apply. 

[ ] Filter raw sensor data (e.g. IDS alerts). 

[ ] Point out the suspicious activities from filtered data. 

[ ] Collect evidence from multiple sources (e.g. IDS, package dumps, etc.). 

[ ] Group individual activities and make hypotheses about an intrusion 

attempt. 

[ ] Assess attacker identity and mission impact 

[ ] Tuning sensors to look for predicted attack 

[ ] Incident handling 

[ ] Produce documents to report current situation awareness 

[ ] Perform virus/incident handling 

[ ] Train others of situation awareness 
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Appendix D: Post-Task Questionnaire 


Subjective Survey # 1 


For the following questions, indicate with your opinion on a scale of 1 through 5. 


1 - Strongly Disagree 3 - Neutral 

2—Somewhat Disagree 


4 - Somewhat Agree 

5 - Strongly Agree 


1) The graphical displays were visually more appealing to the eye than the tabular 
display. 

2) I easily understood the visualization of the graphical displays. 

3) I easily understood the visualization of the tabular display. 

4) The manipulation of the visualization’s features of the graphical displays was 
easy. 

5) The manipulation of the visualization’s features of the tabular display was 
easy. 

6) I was able to identify all of the intrusion alerts on the graphical displays. 

7) I was able to identify all of the intrusion alerts on the tabular displays. 

8) I was able to identify all of the network intrusions on the graphical displays. 

9) I was able to identify all of the network intrusions on the tabular display. 

10) The demo training provided by the investigators enabled me to use effectively 
tbe tool. 

11) 1 was able to complete my tasks better with the tabular display than the 
graphical displays. 

12) I prefer the graphical displays to the tabular display. 

13) I recommend that the use of the graphical displays along with the GUESS 
visualization tool be incorporated into analyst’s cyber-security systems. 

14) 1 recommend that the use of the tabular display along with the GUESS 
visualization tool be incorporated into analyst’s cyber-security systems. 

15) 1 do not recommend that the graphical displays along with the use of the 
GUESS visualization tool be incorporated into analyst’s cyber-security systems 
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16) I do not recommend that the tabular display along with the use of the GUESS 
visualization tool he incorporated into analyst’s cyher-security systems. 

17) The phase two system provides me helpful guidance hy suggesting relevant 
experience pieces. 

18) Most experience pieces suggested hy the phase two system are relevant. 

19) The representation of experience in the phase two system is easy to understand. 

20) Interacting with the phase two system distracts me from concentrating on 
reasoning. 

21) The display of the phase two system helped me manage my hypotheses and 
was very useful. 

22) Generally, the phase two system makes a positive impact on my reasoning 
process. 
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Appendix E: Post-Task Questionnaire 


Subjective Survey #2 

1. Overall, how would you rate the usefulness of the graphical layouts? 

o Excellent 

o Good 

o Fair 

o Poor 

2. Overall, how would you rate the appearance of the tabular layout? 

o Excellent 

o Good 

o Fair 

o Poor 

3. What components of the displays were most effective? 


4. What aspects of the visualizations did you like best? 


5. What aspects of the visualization did you not like? 


6. What aspects of the visualization s helped you to identify intrusions? 
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7. Do you think the phase two system can really help analysts do analytical reasoning in 
cyber analysis tasks? What advantages does it have? 


8. What three things did you like most about your interactions with the phase two system today? 


9. What three things did you like least about your interactions with the phase two system today? 
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Appendix F: Post-Task Questionnaire 


Analysis Survey #1 

Answer the following questions based on the node representations provided in FIGURE 1 below. 
Indicate your selections by writing the letter of the node representations in the blank line. 



Note*(The same representation can be used in multiple answers.) 
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1) From the set (G, H, I, J, O, P), which representation is best suited for representing the 

activity of the system, i.e., top talker?_ 

2) From the set (A, K, L, M, N, P), which representation is best suited for labeling a system? 


3) From the set (C, D, E, F, G, I, J, O, P), which representation is best suited for 

representing the number of users located at a system?_ 

4) From the set (H, I, J, P), which representation is best suited for representing the relevant 

past history of a system?_ 

5) Which representation is best suited to represent an active system?_ 

6) Which representation is best suited to represent an inactive system?_ 

7) Which representation is best suited to represent a system under attack?_ 

8) Which representation is best suited to represent a system that is vulnerable?_ 

9) Which representation is best suited to represent a system that has been compromised? 


10) Which representation is best suited to represent a high priority system?_ 

11) Which representation is best suited to represent a low priority system?_ 

12) Prioritize the following network parameters in terms of relevance to analysis, 1 being 
highest priority, 13 being lowest priority, NA means it is not used/relevant/of interest. 
_CPU Load 

_Number users 

_Number connections 

_Network bandwidth usage 

_% Disk usage 

_% memory usage 

_# alerts generated 

_Type of alerts generated 

_Previous identification of issues with a specific system 

_Median size of packets 

_Connections asymmetry 

_Operating system type 

_System priority 

_Other(s):_ 
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Appendix G: Post-Task Questionnaire 


Analysis Survey #2 

Answer the following questions based on the link representations provided in FIGURE 1 below. 
Indicate your selections by writing the letter of the link representations in the blank line. 



Note*(The same representation can be used in multiple answers.) 
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1) Which representation is best suited to represent a connection from one system to another 

system?_ 

2) Which representation is best suited for representing users with multiple connections? 


3) Which representation is best suited to represent a TCP connection?_ 

4) Which representation is best suited to represent a UDP connection?_ 

5) Which representation is best suited to represent access to a Network File System (NFS)? 


6) From the set (I.. .M), which representation is best suited for representing connections to a 

server? _ 

7) From the set (I.. .M), which representation is best suited for representing connections to a 

client?_ 

8) From the set (I.. .M), which representation is best suited for representing connections to a 

UNIX system?_ 

9) From the set (I.. .M), which representation is best suited for representing connections to a 

Windows system?_ 

10) Which representation is best suited for representing CONUS connections?_ 

11) Which representation is best suited for representing OCONUS connections?_ 

12) Which representation is best suited to represent activity that generated an alert?_ 

13) Which representation is best suited to represent the connection from a system under 

attack?_ 

14) Which representation is best suited to represent the connection to a system under attack? 


15) Which representation is best suited to represent an unauthorized system connection? 


16) Which representation is best suited to represent normal traffic communications between 

systems?_ 

17) Which representation is best suited for asymmetry of connections between inbound and 

outbound?_ 

18) Which representation is best suited for representing the number of connections over the 

past 5 minutes?_ 

19) Which representation is best suited for representing the number of connections over the 

past 1 hour?_ 

20) Which representation is best suited for representing the number of connections over the 

past 24 hours?_ 
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Appendix H: Pre-Task Questionnaire 


Analysis Survey #3 

(Before execution of the study) 

Answer the following questions to the best of your abilities. 

1) On a scale from 1 to 5 where 1 is ‘Very Sad’, 3 is ‘Neutral’, and 5 is ‘Very Happy’, 
which of the following best describes your current emotional state? 

o 1-Very Sad 

o 2-Sad 

o 3-Neutral 

o 4-Happy 

o 5-Very Happy 

2) Which is better at the following tasks, Machine or Human? Write your answer in the 
blank line. 

o Analyzing data_ 

o Detecting anomalies (where an anomaly is an abnormal behavior on a 

security network)_ 


3) 


In general, which type of display do you find most useful in analyzing data? Check all 
that apply. 

o Tables 

o Textual 

o Line & Bar Graphs 

o Simple Graphs 

o Symbolic shapes 

o Other: 


4) If you are not a cyber security analyst, are you interested in what they do? 

o Highly interested 

o Somewhat interested 

o Unsure 

o Not quite interested 

o Not at all interested 

5) What motivated you to participate in this study? 
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6) What do you expect to learn from this study? 
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Appendix I: Post-Task Questionnaire 


Analysis Survey #4 

(After execution of the study) 

Answer the following questions to the best of your abilities. 

1) On a scale from 1 to 5 where 1 is ‘Very Sad’, 3 is ‘Neutral’, and 5 is ‘Very Happy’, which of the following 
best describes your current emotional state? 
o 1-VerySad 

o 2-Sad 

o 3-Neutral 

o 4-Happy 

o 5-Very Happy 


2) Which is better at the following tasks, Machine or Human? Write your answer in the blank line, 
o Analyzing data_ 

o Detecting anomalies (where an anomaly is an abnormal behavior on a security 

network)_ 


3) 


Which types of displays do you find most useful in analyzing security data? Check all that apply, 
o Tables 

o Textual 

o Line & Bar Graphs 

o Simple Graphs 

o Symbolic shapes 

o Other:_ 


4) If you are not a cyber security analyst, how likely now are you to become one? 
o Very Likely 

o Likely 

o Unsure 

o Not Likely 

o Very Unlikely 


5) 


If you are a cyber security analyst, how likely are you to be one in the future? 
o Very Likely 

o Likely 

o Unsure 

o Not Likely 

o Very Unlikely 


6) What features of the visual displays were most useful for completing your tasks in this study? 


7) What did you learn from this study? 
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Appendix J: Email recruitment letter for ARE analysts 


Good morning/afternoon, 

The U.S. Army Research Laboratory (ARL) Computational and Information Sciences Directory 
(CISD) and Human Research and Engineering Directorate (HRED) are seeking adults (ages 18 
and above) with cyber security network analysis experience to participate in a research study 
evaluating effective visual displays for analysts. In the research, we employ computer graphical 
displays to represent computer network activity that network analysts currently view in a tabular 
format. We are interested in participants’ response to a simulated cyber-security analysis game 
scenario. During the study, participants act as analysts and their job is to identify as many of the 
intrusion attacks and intrusion attack attempts on a simulated network using tabular and 
graphical displays. We will use the results from the study to help understand which of the visual 
layouts is most effective for data analysis prediction. This new insight is beneficial for network 
security analysts tasked with defending the nation’s networks from cyber attacks. 

If you elect to take part in the research study and are an employee of ARE, you will participate 
during your regular tour of duty for a maximum of 1 hour per day during a maximum of 5 days. 
We expect the study to take 2-3 hours for most participants, depending on how quickly tasks are 
completed and how many rest breaks are taken. There is no compensation or personal benefit for 
your participation in this study. The study will take place on Aberdeen Proving Ground (APG) in 
Building 459. Transportation will be provided from other locations at APG, and gate access will 
be coordinated prior to the study. You can withdraw from this study at any time. Even if you 
come to the research site and start the study, you can change your mind and withdraw from the 
study without penalty. 

If you would like additional information, please contact the principal investigators: 

Renee Etoty 

Network Security Branch, ARE Computational and Information Sciences Directorate 

Building 204, Room 2D068, Adelphi Eaboratory Center, MD 

(301) 394-1835 

renee. e. etoty .civ @ mail. mil 

Dr. Robert Erbacher 

Network Security Branch, ARE Computational and Information Sciences Directorate 
Building 204, Room 2C100, Adelphi Eaboratory Center, MD 
(301) 394-1674 
robert.f.erbacher.civ@mail.mil 


Project Number: ARE 13-050 
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